On the Integration of Structured Data and Text: A Review of the SIRE Architecture (invited talk)

1.0 Introduction Over the past decade, members of the Information Retrieval Lab have designed, developed, and deployed a variety of information retrieval systems. A central theme for all of our systems was the integration of structured data and text. One of our more recent efforts, SIRE, a Scalable Information Retrieval Engine [Grossman97, Grossman98, Lundquist99] is the focus of this paper. For completeness, I review some of the functionality of SIRE although it is described, in greater detail, in other forums. We describe the architecture of the prototype developed for the National Institutes of Health (NIH) National Center for Complementary and Alternative Medicine (NCCAM) [Frieder00] by some of the members of the laboratory. The version deployed at NCCAM is a more industrialized version of this prototype. The mainstream approach in the development of information retrieval systems uses a customized inverted index to represent the text. SIRE, on the other hand, is a relational information retrieval approach and uses relations to model an inverted index. Storing the full text in a relational environment integrates the search of unstructured data with the traditional structured data search of Relational Database Management Systems (RDBMS). By using only standard SQL, SIRE leverages investment of the commercial relational database industry, while providing all capabilities of a more traditional information retrieval approach. RDBMS offer a wide variety of functionality such as concurrency control, recovery, security, portability, scalability, and robustness. RDBMS vendors continuously improve these features and incorporate advances made in hardware and software. Thus, an application using an RDBMS is able to keep up with the technology curve with less investment than a custom solution. SIRE is implemented strictly as a relational database application. Key information retrieval techniques such as leading similarity measures, proximity searching, n-grams, passages, phrase indexing, and relevance feedback are all implemented using standard SQL. By adhering to strictly standard SQL, SIRE is completely portable across platforms and database management systems. Thus far, either laboratory members or our industrial collaborators or sponsors implemented SIRE on the NCR DBC-1012, Microsoft SQL Server, Sybase, Oracle, IBM DB2 and SQLD/S database management systems. SIRE achieves good performance and scalability and is currently in production use in a variety of text search applications both commercially as part of multiple industrial efforts and in various government laboratories including the National Institutes of Health National Center for Complementary and Alternative Medicine. The relational platform offers equivalent capabilities to traditional Information Retrieval (IR) systems while providing a parallel, scalable, maintainable architecture with a unified platform for integrating searches of structured and unstructured data.